Computer Security is Not a Science ( but it should be )

Security research is sometimes referred to as the “Humanities of Computer Science” because, too frequently, “secure” systems are built using equal measures of folklore and black arts. Despite the humorous intention, there is a kernel of truth in this jest— computer security, at least “security in the large”, is not currently a science. This claim may seem unfair, given the progress made in security over the past decades. However, our present tools and methodologies are at most adequate for understanding systems security on a small scale. Cryptography, for example, is perhaps the most thoroughly studied and most rigorously modeled aspect of security. Despite its tremendous importance, cryptography alone is not sufficient for building secure systems. Indeed, the vast majority of all security flaws arise because of faulty software (e.g., the ubiquitous buffer overflow problem). Such security holes cannot be avoided by cryptographic techniques, and despite widely known and accepted solutions to these kinds of software flaws, buggy code persists. Why is security not a science? Some would argue that, by nature, security is fundamentally unscientific: security is hopelessly intertwined with social and economic forces beyond the purview of science. Yet, economists and psychologists have developed testable, scientific theories. What sets science apart from other disciplines is that it produces hypotheses that can be experimentally verified (or falsified). But, despite the large amounts of security-relevant data collected by organizations like CERT and despite our decades of experience building systems, computer security research has produced little in the way of predictive models or experimentally verifiable hypotheses. How can we establish security in the large on a more scientific footing? Over the last millenium, one way that disciplines have evolved into “sciences” is through a period of quantification. For example, Galileo, among others, transformed physics from an Aristotelian philosophy to a Baconian science by describing distance, speed, and time quantitatively, rather than explaining why objects fell, rolled, or flew. Our belief is that the current pre-scientific state of security research is fundamentally due to a lack of reasonable metrics. Furthermore, although there exist a few experimental methods for assessing security (i.e., tiger-teaming [5]), these methods are not yet particularly meaningful in the context of science, where quantitative evaluation—for comparison, modeling, and measurement of achievement—is central. The main questions we are interested in addressing are: Question 1: How could one measure security quantitatively? Question 2: What experiments ought one perform to assess security? Question 3: How can we improve our models using these metrics? We believe that we can eventually achieve a workfactor-like formulation to address the first question. Such a formulation will likely be a composite of a variety of measurements, with imprecise but meaningful weights. As in physics, an approximation that can gradually be refined with experience is very use-